We report on a search for gravitational waves from the coalescence of compact binaries during the third and fourth LIGO science runs. The search focused on gravitational waves generated during the inspiral phase of the binary evolution. In our analysis, we considered three categories of compact binary systems, ordered by mass: (i) primordial black hole binaries with masses in the range 0.35M center dot m(1), m(2) 1.0M center dot, (ii) binary neutron stars with masses in the range 1.0M center dot m(1), m(2) 3.0M center dot, and (iii) binary black holes with masses in the range 3.0M center dot m(1), m(2) m(max) with the additional constraint m(1) + m(2) m(max), where m(max) was set to 40.0M center dot and 80.0M center dot in the third and fourth science runs, respectively. Although the detectors could probe to distances as far as tens of Mpc, no gravitational-wave signals were identified in the 1364 hours of data we analyzed. Assuming a binary population with a Gaussian distribution around 0.75 - 0.75M center dot, 1.4 - 1.4M center dot, and 5.0 - 5.0M center dot, we derived 90%- confidence upper limit rates of 4.9 yr(-1)L(10)(-1) for primordial black hole binaries, 1.2 yr(-1)L(10)(-1) for binary neutron stars, and 0: 5 yr(-1)L(10)(-1) for stellar mass binary black holes, where L-10 is 10(10) times the blue-light luminosity of the Sun.