Sakuras are not fragile by definition. The problem comes from the population density of these trees, which causes certain fungus infections can infect a lot of trees at once. These fungus can live indefinitely on the tree bark without creating any symptoms on the infected trees, but any open wound, like, snapping a branch or so, can create entry point of which it can infect the entire tree and causes it to wither.
It's the same problem with the current mono-variety banana tree farms.
That being said, a single puchi breaking a single twig off isn't going to spell doom for the tree right away as most Sakura trees likely get damaged multiple times over the course of a viewing season what with all the drunk blue collar workers around. It's just that the damage increases by many magnitudes once people get used to cavalierly plucking a souvenir.
The heck are you two doing...?*Snap!*Nya~ ♪Nyaha~If you break off part of a cherry tree, germs will enter from there and ruin themーI-I'm not going to break anything!Bwah!Nyaa...
Nyaa nyaa...Nyaa...Nyaa...*Tremble**Thumbs up**Tremble**Tremble**Tremble**Hop**Hop*