How to remove adhesive   

Have you tried just soaking it in water, and then using your fingernails to take the glue off, or maybe just putting it through the wash cycle?

This is what Id do. Id use lighter fuel and dab it very lightly and sort of rub at the same time. Now, be sure you're not smoking while doing this and you'll need to wash your sweater after. Its always worked for me. Hope it'll work for you.

If you let some Goo Gone sit for a few hours on that part of the sweater, it should get it off. Then wash the sweater, because once washed, you'll never be able to tell Goo Gone was there in the first place. OR, put an ice cube on it, let it set until it starts to melt, and then see if it comes up.

carmelanirel did you try googling it to find out how to get rid of
fabric glue, Sometimes hair spray will remove chewing gum so it might
work on fabric glue but I would test first on inside seam of sweater for any adverse action on the acrylic sweater material, I am going to
yahoo and mylot search to see if there is some good way to do this.

You can try peanut butter, peanut butter works on a lot of sticky things, it shouldn't stain but you might want to try it on an unseen area first to make sure, if it seems as if it will wash easily then saturate the area with peanut butter and then just let it set on there for a while maybe like an hour, the oils in the peonut butter will usually break up the sticky stuff, you may need to do it more than once.

It will be a tedious job at best. Try eucalyptus oil, it's good for removing all kinds of things.