Yes!
H2SO4 is a very strong acid, that means it dissolves completely.
So 1 molecule of H2SO4 dissolves into two H+ molecules (this is the important thing to note).
pH is equal to -log(c(H+)) so you aim for a concentration of H+ ions of 0,001 mol/L(pH = 3) and 0,0001 mol/L (pH = 4). So you need 0,0005 mol/L (pH = 3) and 0,00005 mol/L (pH = 4)H2SO4 in your solution. Since you want 100ml it is divided by 10 again. Then you need the molar mass of H2SO4 to get the mass you need to add. 1 mol H2SO4 = 98.1g
So for 100 ml Water with pH = 3 you would need 5mg H2SO4 for pH = 4 you would need 500ug.
As you can see this is not easy to do without an analytical scale.
I'd suggest you made 10 litres then you need to put in 500mg or 50mg.