Filler-induced strengthening is ubiquitous in material science and is particularly well-established in polymeric nanocomposites. Despite having similar constituents, colloidal gels with solid filling exhibit distinct rheology, which is poorly understood. We show using experiments and simulations that filling monotonically enhances the yield stress of colloidal gels, while the elastic modulus first increases before decreasing. The latter effect results from a disturbed gel matrix at dense filling, evidenced by a growing inter-phase stress. This structural frustration is, however, not detrimental to yielding resistance. Instead, fillers offer additional mechanical support to the gel backbone via percolating force chains, at the same time decreasing the yield strain. We develop a mechanistic picture of this phenomenology that leads us to a novel `filler-removal protocol,' making possible individual control over the strength and brittleness of a composite gel.