The mailNickname is generated from the URL settings of your request.
The {EncodedTitle} tag will remove a range of special characters but not all can be covered. If you are using {Title} this will pass through spaces and special characters and will cause the failure.
This URL is used to set the Site URL and the Group mailbox email, this cannot contain special characters or spaces.
If any invalid characters are included on the request, it will produce the error: Invalid value "mailNickname" of resource "Group"
Resolution
In this case the request cannot be replayed as the erroneous character cannot be removed.
Delete the request from the Job Queue and ask the requestor to make the request again without the special character.
If you have copied the name from another location the special characters may be hidden, please manually re-type the name in this case and try again.
Information from Microsoft: