The in game counter is character specific while grimoire is account wide. If there is a difference in the two, it means you have collected fragments on another character. You'll have to pick up the ones you grabbed on the other character