As said in the comments, a UDF is a function that has to return a valid cell value. Returning an object or doing any object manipulation will result in an error.
You could easily fix your problem by splitting the two tasks:
First, have a UDF as you had before, just remove the fill part.
Instead, use public variables to store the local ones passed to the function in:
Public r As Integer
Public g As Integer
Public b As Integer
Public Function AssignColor(r1 As Integer, g1 As Integer, b1 As Integer) As String
r = r1
g = g1
b = b1
AssignColor = "#" & Application.WorksheetFunction.Dec2Hex(RGB(r, g, b))
End Function
Next, in the worksheet module, hook into the worksheet change event and use the public variables to do the color manipulation after the UDF has been fired. This way you avoid the limitation. Remember that the coloring happens in response to someone having entered that UDF into the cell, not as part of calculating the formula result.
Private Sub Worksheet_Change(ByVal Target As Range)
'Only change cells holding our formula and avoid case sensitiveness.
If LCase(Left(Target.formula, 12)) = "=assigncolor" Then
Target.Interior.Color = RGB(r, g, b)
End If
End Sub
Hope this helps.