The title says it all.

Which compounds are responsabile for its smell?
And what compounds are generated during its vaporization?

What exactly do you smell? It should be mostly DMT, derivatives and byproducts of the former.

Dmt smells like alien flowers of a sneaker variety. I had some dmt that tasted like a rotten tooth, which told me dentists will never get down with this substance. Too this day, never found a dentist that got down with the dmt. Coincidence, I think not.

Although I also get that new sneaker smell from DMT, I also associate it with jasmine. The scent of jasmine contains indole which is essentially any tryptamine stripped of all branches, just the two rings. It is a complex scent which no doubt contains other derivatives of both indole and tryptamine. I don't know, but would suspect that 100% pure DMT would have no scent. Although, this may not be possible when exposed to oxygen. Just 0.001% indole would produce a very noticeable jasmine scent and the same would be true for many of the other impurities that would accompany both extracted and synthesized DMT.

I suspect skatol (3-methyl indole) could have been the secret ingredient in your rotten tooth DMT. I have some experience with a failed synthesis that resulted in a fair amount of skatol. It carries a pretty potent stench. A certain room smelled of feces and rot for a couple weeks.

The smell comes from DMT and its decompostion products.

The ones that smell the most are the both below.
Indole (without the CH3) smells like flowers

Skatole (with the CH3) smells like feces/disgusting.

Pure indole has a strong flower smell but even the slightest impurity of other decomposition products like skatole turns it pretty bad.

Same counts for dmt.

Indole and its derivatives has a good/bad smell for us humans because tryptophan in our body decomposes to indole and scatole which is usually the smell of death/feces/infection.

But because its found in flowers it smells good too for some people if pure.

Here is a table of many indole compounds. If you click you can see properties like smell for most of them. Most have feces floral/jasmine.

I remember smelling some very brown DMT for the first time and auomatically associating it to "rotten organic".

Smells and tastes like rubber/plastic/mothballs/flowers to me.

I think residue in my pipe smells like an old mans bad breath. But fresh crystals smell like sweet flowers from another universe

Absolutely agree.

Its all bout the storage.
Next time you have some "fresh" flower dmt take a small bit and keep it somewhere in contact with air and sun.
It will no longer be flower after a few days