Solution 1
We claim that the only solutions are
and
. These clearly work.
First note that if there exists some
such that
, then we can immediately conclude that
is linear by placing
, which leads to the two above solutions. We will now assume for the sake of contradiction that there does not exist such a
:
By letting
, we obtain that
. Since
by the above assumption, we see that there does not exist an
such that
.
Therefore, given any
, we have that
is not in the range of the quadratic
, which means that

With this bound, we can now control the size of
quite well:
Let
so that
. Thus:

Now if we let
tend to infinity in
we can see that
. This leads us to a contradiction, since then as
approaches infinity, the LHS in
is bounded between
and
(which are constants), while the RHS approaches negative infinity!
Solution 3
The only solutions are
and
, which clearly work. Now we prove they are the only solutions. Suppose there existed another solution. Let
be the given assertion.
Claim:
is not linear.
Proof: Suppose
was linear (and keep in mind we are asssuming
is not the identity or
). Since
is clearly not constant, it must be injective as well. Let
.
gives
, so since injective,
.
, so setting
gives
, so
, so
or
. But these give
and
respectively, so
or
, absurd. 
Claim:
is not in the image of
.
Proof: If
was in the image of
, fix
with
. Setting
gives
, so
is clearly linear. 
Claim: If
is in the image of
, then anything less than
is in the image of
as well.
Proof: Let
.
gives that
is in the image of
. Let
. Since
gets arbitrarily small (negative leading coefficient) and
is a root of
, by IVT,
is surjective over negative real numbers. 
The two claims combined implies that
is negative for all
.
Setting
gives
, so
must hold. Now let
.
We have
![\[ xy + f(x+y) = f(x + yf(x)) + y < x + y + y f(x) \implies g(x + y) > y g(x) \]](//latex.artofproblemsolving.com/3/4/9/3496becec66e41590331a9bcf356ffc865fa664d.png)
Setting
gives that
for all reals
, so for all
,
![\[ g(x) < \frac{g(c)}{c - x} \]](//latex.artofproblemsolving.com/0/1/e/01e9e5cbfd2cf02ac3fab737945a295f78d89410.png)
Thus (as
is always positive),
. From
, we have
. Taking
to negative infinity gives that
![\[ \lim_{x \to -\infty} f(x f(0)) = 0 \implies \lim_{x \to \infty} f(x) = 0 \]](//latex.artofproblemsolving.com/4/9/6/496cff1b3e88dcf47e069680f9e38ce157a4ae64.png)
because
is negative.
. Setting
gives that the RHS approaches infinity, but the LHS is always less than
, a contradiction.