Hi there "jordansvoice", and Welcome to the forum!
All I can say initially is: "Wow!" It's pretty obvious you are rather frustrated, to put it mildly, and you seem to be chasing a chimera that you can't quite seem to grasp: it's always a few steps ahead of you.
I just listened to your samples, but it tuns out the the first two are identical! Maybe you can check that, and add the missing one?
OK, so your basic question is : "How in hell do I fix this damn thing?". Answer: I would suggest taking a big step back, and looking at this logically, from the beginning, using the laws of acoustics, not the laws of emotions!
First, you should understand that by the simple act of isolating the room so well, you have prevented sound from getting
out! That's what isolation does, and it's this flip side that most people don't think about when isolating. As you said yourself: "Sound does NOT get into this booth". But for the same reason, it also does not get out! Isolation stops sound from getting out, so therefore it must stay in.

Logical. And since it this one is isolated much better than your last place, it keeps it in much better. In that last room, most of the low frequency sound was getting out and disappearing, but now it is sticking around, reverberating (not resonating), and messing things up. So the problem you are experiencing is caused, in part, by the greatly improved isolation.
Second, the room is a resonant box: All rooms are. It will resonate (not reverberate) at certain frequencies. There might or not be a good relationship between those frequencies. Also, resonance and reverberance are two different but related things.
So there are resonant issues, and reverberant issues. And there are also different
modes of resonance, which is likely where your issue lies: However, it might also be related to the boundary interference response of the room, which is something else entirely, but can look the same (or rather, can sound the same).
Third, the room is not rectangular, so there is no simple way to predict how it will behave. It is actually two connected resonant spaces, one with a higher ceiling and the other with a lower ceiling, but to accurately predict how they interact together is rather complex. If there was just one single ceiling height it would be simple...
So the first order of business is to figure out which
specific problem(s) you have, and which
specific frequency(ies) is/are involved. Based on that, you can decide how to treat the room.
Unfortunately, you cannot determine this by talking and listening inside the room. The only way to do it, is by running an acoustic analysis on the empty room, without any treatment in it.
Fortunately, that is easy to do: Start by downloading REW, which is an amazingly power acoustic analysis tool, that also happens to be free! You can get it from the Home Theater Shack website. Install it on a computer with a decent soundcard in it, hook up your mic in Omni mode, and a good full-range speaker, and run some tests without any treatment in there. REW will reveal the entire ugly truth about the room, in terms of frequency response, phase response, impulse response, spectral response, and most important of all, the time energy response ("ETC"), which basically means how the acoustic energy inside the room decays over time, across the entire spectrum.
When you are set up with the gear to do that, then let me know and I'll walk you through the process. In your case, you'll be doing it "backwards" from the normal method, since your room is a vocal booth where you are recording sounds, not a control room where you are playing them back. REW is really meant for the later (playback), but can still be used for the former, just with a different procedure. Part of that involves moving the mic and speaker to a series of different locations inside the room, and running a test at each location. That will help determine if you have a modal problem (it will stay roughly the same at all locations). or an SBIR problem (it will move to different frequencies at different locations), or a comb filtering (phase cancellation) problem (moves around the spectrum for different locations), or a some combination of these (which is the most likely case). Or perhaps there is even some other issue.
If you do the REW test right, you will be able to identify the exact set of problems that you have, which are combining to cause the overall effect that you are hearing.
So download REW, set up your gear, remove all the acoustic stuff you have in there at present to get down to bare drywall again, do the calibration procedures listed in the REW manual, and let us know when you are ready to start the sequence of tests.
In the meantime, here are some general comments on the rest of your post, that might be helpful, plus some questions:
So I thought I was brilliant and decided to use the much more affordable Ultratouch Denim Insulation. A mixture of R-13 and R-19. The NRC rating for 125hz was over 1.0 so I thought it was a no brainer.
Do you have the acoustic specs for the specific stuff you used, as published by the manufacturer? If so, could you post it here, or post a link to it?
Also, there seems to be some confusion here: the NRC rating is a single number that rates the
overall absorption of the material: it is not the absorption in individual frequency bands. Those are normally shown in a table of "absorption coefficients", with the NRC rating sometimes given in the last column of the table. There aren't many materials that have an NRC of 1, and materials that have a coefficient of one at 125 Hz aren't that common either (there are a few). Absorption coefficients also depend on the thickness of the material, and the mounting method, and the testing method can also lead to readings that are too high in some cases, so just because you see a "1" in the table does not mean you will actually get that in real life.
I have very very good ears that I trust and are nearly always right when it comes to this kind of stuff.
I don't doubt that your ears are great, and you have excellent pitch sense, but the human auditory system can never be anywhere near as accurate as a proper acoustic test. There are psycho-acoustic issues that you cannot compensate for, and will never be able to distinguish accurately, no matter how good your ears are. Mics, speakers and acoustic software does not suffer from those issues, which is why you need to run the tests, so you can see the graphs of the things that your ears cannot hear.
After doing a lot research I realized that the denim just wasn't DENSE enough to absorb the lows!
You probably have that backwards! It is a myth that higher density porous absorption is better for low frequencies. In fact, the reverse is true: lower density porous absorption is better for low frequencies, and higher density is better for highs. So it is highly probable that our material was
too dense, not that it wasn't dense enough!
In fact, it's not even the density that matters. What REALLY matters, is a characteristic of the material known as "gas flow resistivity", which is measured in the unlikely and obscure units of Rayls. Gas Flow Resistivity ("GFR") is all about how the material reacts to gas moving through it: Air is a gas. Sound waves make air molecules move. Bingo!

You see the connection. If you know the GFR number for a specific material, then you can figure out all you need to know about how it will impede sound as the sound waves move through it. (OK, not really "all", but "a hell of a lot"). Unfortunately, most makers of common insulation materials never bother measuring GFR, or if they do, they don't publish it. For a simple reason: it isn't very important at all for understand the thermal properties of the material, which is what they really want to know! Only manufacturers of acoustic products bother measuring it. So you'd probably find it hard to come up with the GFR number for your denim insulation, unless the manufacturer specifically sells it as an acoustic product, not a thermal product.
Fortunately, it turns out that there is an approximate relationship between the GFR of a specific type of insulation, and its density. The relationship isn't linear, and is different for each type of material, but it's good enough to get an idea of how a particular product will behave acoustically, even if the manufacturer never tested it for acoustics.
So if you know the type of material, and the density, then you can roughly figure out the GFR, and hence the absorption characteristics... to a certain extent. For example, it turns out that fiberglass insulation with a density of 30 kg/m3 has roughly the same GFR as mineral wool with a density of 50 kg/m3, so they could both be used for the same purpose. But mineral wool with the
same density as the fiberglass, would not perform the same.
The problem here is that, unless you now the relationship for your denim insulation, there's no way of telling just how dense it should be!
I even tried bringing in many 2 to 4" thick fiberglass panels from my old booth. It cut the lows by maybe, 10%.
10% is a subjective guess, and please don't get me wrong here, but it is basically meaningless. A 10% reduction in sound intensity is a reduction of less than half a decibel (0.46 dB, to be more precise), which is inaudible to the human ear. If you heard a very slight change in sound intensity that you could just perceive, then that would be a change of 2 or 3 dB, which is a change of about 50% to 60% in actual sound intensity. Even though our brains might want to assign percentages to sound differences, we can't really do that, since percentages are linear and ears are logarithmic. Our brains are not very good at making the connection, so it is much better to always talk in terms of decibels, which are logarithmic already. You might think that you heard a 10% change, but you didn't: that's subjective, and incorrect. I'm not questioning your ears or your ability to judge sound in the least! I'm just pointing out that the terminology you are using, actually has no real meaning in acoustics: You are using linear numbers to refer to a logarithmic event. Not valid.
I bought boxes of Knauf Ecos panels, 3pcf.
I think you mean "Knauf Ecose" panels? Those seem to be rigid fiberglass boards, and that is a bit too dense for good low frequency absorption. Which is why it is not working the way you hoped. Which specific product did you buy? (A photo of the packaging would help)
But the corners are causing bass buildup, not the flat/middle part of the ceiling.
right. That's always the case, in any room. All room modes terminate in the corners. The center part of the ceiling/wall/floor is more commonly the source of flutter echo, and only partly related to bass build-up.
After weeks and weeks of testing. I've determined that the bass buildup problem is EXCLUSIVE to "higher" ceiling in my booth. Speaking underneath the shorter ceiling does not produce boominess.
That's where you are HEARING the problem, but that might not be what is CAUSING the problem. Two different things. The point in the room where you hear it loudest is likely either a modal peak (node) or an SBIR peak (I'm guessing modal, not SBIR), but what is
causing it is simply the relationship between the various dimensions of the room. A mode is just a path that a sound wave of a certain frequency can take around the room, bouncing off two or more walls along the way, and arriving back at its starting point while going in the exact same direction, and in phase with itself. It's a bit more complicated than that, but it helps to think of it like that. A mode might involve 2 room surfaces (walls, floor, ceiling, etc.), or four, or six (or even more, for complex modes in a room with more than six sides, such as yours).
A mode is a standing wave, and while the peak might be located under the high ceiling, the mode might actually be due to room surfaces that have no relationship at all to that ceiling. You cannot assume that just because you hear it at under the high part of the ceiling, that it is being caused by that part of the ceiling. By the same token, if that ceiling is not involved in causing that mode, then it doesn't matter how much absorption you pile onto it, it won't have any effect at all, and instead will just make the room sound bad in other ways, because it is absorbing too much of the high end of the spectrum, while not addressing the actual problem at all.
So, I purchased a bunch 4" thick, 4' x 2' 6pcf Roxul Rockboard sheets.
Those are mineral wool: different GFR characteristics. Not necessarily comparable to the Knauf stuff. It will likely not react the same. And in any case, 6 pcf is way too dense for bass trapping: that's nearly 100 kg/m3! That will definitely kill the highs beautifully, but won't do to much for the lows....
When I speak while sitting down or kneel, the boominess is drastically reduced.
Actually, it isn't reduced at all: you are likely just speaking from the location of the modal null for the specific mode that is giving you the most problems, and since you are in the null, your voice won't trigger that mode, or will trigger it at a much lower level. So the boominess didn't go away: you just moved your mouth to a position where it isn't.
I've ran bass pink noise through a monitor, scanned with an SPL meter and confirmed it's the high ceiling corners resonating like mad.
Once again, all that you really confirmed is that you can hear and measure the room modal response, which is ALWAYS in the corners, because that is where ALL room modes begin and end! What you have discovered is sort of like saying that you went into the kitchen, put your hand in the flames over the gas burner, and discovered that this is the hottest place in the kitchen: Of course it is hot there, because that's where you are burning the gas! Of course there is bass energy build up in the corners, because that's where the modes all meet! But that information is not useful, in both cases, because it is already known.
But the fact that you can see a bigger number when you put your meter in the corner is totally unrelated to the problem that you have in the room: You will find exactly the same energy increase in ALL corners of ALL rooms, just as you will find that the hottest place in ALL kitchens, is the flame over the burner. But measuring the temperature of the flame doesn't tell you anything about how well you can cook in that kitchen, just the same as measuring the sound level in the room corners does not tell you anything about how well you can record in that room. You are looking at the wrong things, and drawing the wrong conclusions.
Please reference the images below for the internal dimensions of the booth.
With those dimensions, ignoring the soffit at one end of the room and assuming it is a rectangle, your first three axial modes are at 80 Hz, 97 Hz, and 113 Hz. You say you can hear issues in the area o 125 to 150 Hz, but you only have three tangential modes in that region (126 Hz, 138 Hz, and 149 Hz). It's not so likely that those are the problems, so I'd hazard a guess and say that the real issue is the second harmonic of your longest axial mode, at 160 Hz, as well as the fundamental height mode at 113 Hz.
Of course, that's just a very rough "guesstimate", without taking into account the soffit, so it could be way off the mark. Only REW will reveal the truth!
Question: How in the hell do I eliminate this crazy bass buildup in the ceiling?
First: measure the room with REW. Then analyze the results. THEN decide how to treat the issues that REW shows.
The fact that only some of the bass is reduced during my rough "hold the 4" 6pcf panel up and talk" test makes me feel very boggled and hopeless.
That's most likely because you are using the wrong product in the wrong place.
My 2 proposed solutions, but please, give any other BETTER ideas that you think of:
1) Use the 4" thick 6pcf Rockboard panels to create a super chunk trap running the full length of the ceiling.
I would not do that. As I said before, 6 pcf is too dense, and you don't even know for sure if the ceiling is involved in creating the problem. It quite possibly
is involved, but you'll never now until you test the room. Treating a surface that is not involved in the problem, is not going to help.
2) I have a feeling that I need to mix layers of different absorbent materials because this bass is just plowing through dense, rigid, insulation.
Mixing layers is unlikely to help. The reason the bass is "plowing through dense, rigid, insulation" is because that's what bass does!

If you were to use less dense, lighter, less rigid insulation, you'd probably get better results.... but only if you put it in the correct location!
So I was thinking a layer or two of the 2mm Audiomute Peacemaker rubber applied to all of the ceiling drywall/corners.

... Ummm ... NO! Most definitely NO! Rubber is NOT the solution. (Unles you feel like trying to make a limp membrane absorber that is tuned to the modal frequency that you need to treat... and good luck with that if you do decide to try! The box would take up half the room, for such a low frequency....).
Then, in the airgap behind the straddling panels, slide in a layer of FRK along with my old Ultratouch Denim. 4 total different materials that do absorb lows in some fashion.
That actually WOULD be a membrane trap! What frequency would you tune it to, and how would you go about tuning it?
I'm not worried with how much high's/mids are there right now as I'm not done. I need to resolve this bass issue first because
Actually, you need to solve them both together, not one at a time. Fixing bass issues is very, very probably going to cause major issues with your highs, unless you take that into account at the same time. Acoustically dead spaces are very uncomfortable to work in, and a lifeless voice is not likely to bring business rolling in.... Your acoustic treatment has to be balanced, not mangled in piece by piece.
Sorry to be so emotional.
Not a problem. I hear your pain, and the only way to "stop the hurt" is to go back to square one, and start again, logically, intelligently, taking it step by step: run acoustic tests on the empty room every which way possible, analyze the results, identify the REAL issues (not the imagined ones), and treat each of those in turn, with a balanced approach that also takes into account other potential issues. Repeat the process of "measure, analyze, design, install" as many times as necessary to get the room to a usable state.
There's one final point that I just noticed on re-reading your post: "
... in my former booth (which was much larger)".
Hmmmm... since this one is "much smaller", it can never sound as good as the other one: That's a simple fact of acoustics. The smaller a room is, the worse it sounds simply because it has less modal support at low frequencies, more comb filtering in the mids and highs, and less surface area for treatment. So you should start out from the basis that this room cannot be made to sound as good as the other one, due the limitations imposed by the laws of physics. That doesn't mean that it is unusable! It can probably still be improved considerably, and hopefully the improvement will be sufficient that the room is usable for what you want to do with it, but you should also be aware that it simply cannot be made to sound as good as your old room, did, no matter how much treatment you put in.
Anyway, do let us know when you are ready to run the REW tests, and I'll walk you through that.
- Stuart -