Hello! I'm in the process of training a model on a top-40s dataset using your library. However, I want to experiment with long-term consistency, so I've scaled sample rate/channels accordingly to fit ~90s windows during training. I think my results could be improved by further scaling up the number of model parameters, but I'm not sure what to change and by what ratios to get the most bang for my buck/VRAM/compute. Do you guys have a "scaled" config you could share or a general guide (e.g., 2X attention heads, 1.5X mults) for this? Thanks!

Hey there, I have been following this project pretty closely...looks great. Could you share the lightning trainer you're using here/any associated scripts for training?

I've put together my own trainer with accelerate which is working fine, but would be nice to work out of the same one as you here for reproducibility sake.

The loss suddenly increases from <0.1 to billions over one or two epochs.

I'm training an AudioDiffusionModel and I've had happen with both the default diffusion_type='v' as well as with diffusion_type='vk', also, it happens both with and without gradient clipping. It's happened with several datasets and different batch sizes (the output below is a particularly small dataset with a large batch size)

It seems to happen more often, the closer it gets to 0 loss.

Output:

1328 Loss : 0.0562
100% 6/6 [00:01<00:00,  3.93it/s]
1329 Loss : 0.0517
100% 6/6 [00:01<00:00,  3.95it/s]
1330 Loss : 0.0500
100% 6/6 [00:01<00:00,  3.95it/s]
1331 Loss : 0.0374
100% 6/6 [00:01<00:00,  3.93it/s]
1332 Loss : 0.0519
100% 6/6 [00:01<00:00,  3.69it/s]
1333 Loss : 0.0557
100% 6/6 [00:01<00:00,  3.47it/s]
1334 Loss : 0.0499
100% 6/6 [00:01<00:00,  3.33it/s]
1335 Loss : 0.0482
100% 6/6 [00:01<00:00,  3.74it/s]
1336 Loss : 1.4608
100% 6/6 [00:01<00:00,  3.89it/s]
1337 Loss : 35551447.3009
100% 6/6 [00:01<00:00,  3.91it/s]
1338 Loss : 17436217794.0833
100% 6/6 [00:01<00:00,  3.86it/s]
1339 Loss : 15120838197.3333
100% 6/6 [00:01<00:00,  3.88it/s]
1340 Loss : 1137136360.0000
100% 6/6 [00:01<00:00,  3.83it/s]
1341 Loss : 184102040.6667
100% 6/6 [00:01<00:00,  3.80it/s]
1342 Loss : 24171988.5000
100% 6/6 [00:01<00:00,  3.85it/s]
1343 Loss : 100907.1549
100% 6/6 [00:01<00:00,  3.80it/s]
1344 Loss : 10494.4541
100% 6/6 [00:01<00:00,  3.83it/s]
1345 Loss : 989.2273

The model:

class DiffModel(nn.Module):
    def __init__(self):
        super().__init__()
        self.model = AudioDiffusionModel(in_channels=1, diffusion_type='vk', diffusion_sigma_distribution=VKDistribution())
        self.optimizer = torch.optim.AdamW(list(self.model.parameters()))

    def train(self, x):
        self.optimizer.zero_grad()

        loss = self.model(x)
        loss.backward()

        clip_grad_norm_(self.model.parameters(), 1.)

        self.optimizer.step()

        return loss.item()
    ...

Training:

for epoch in range(load_epoch + 1, MAX_EPOCHS):
    acc_loss = 0
    for x in tqdm(dataloader):
        x = x.to(device)
        acc_loss += model.train(x)
    loss = acc_loss / epoch_steps
    print(f'{epoch} Loss : {loss:.4f}')
    ...

Setting the num_steps in DiffusionSampler to 1 leads to nan values when using a UNet. I tried replacing the UNet with an identity function and I do not get the same nans.

Hello,

I'm trying to get the colab to run with the new larger model available here: https://huggingface.co/archinetai/audio-diffusion-pytorch/resolve/main/audio_975.pt

I've modified the colab to use the latest git code: !pip install -e git+https://github.com/archinetai/[email protected]#egg=audio-diffusion-pytorch

However loading the latest model fails with Can't get attribute 'CrossEmbed1d' on <module 'audio_diffusion_pytorch.modules' from '/usr/local/lib/python3.7/dist-packages/audio_diffusion_pytorch/modules.py'>

Is there a way to get the new model to run?

hello, I am new to neural network models, I would like to ask if I am training the model correctly? here is the part of the code

model = AudioDiffusionModel(in_channels=1).to("cuda")
optimizer = Adam(model.parameters(),lr=0.0001)

for i in range(epochs):
  for x in iter(Data):
    loss = model(x)
    loss.backward()
    optimizer.step()
    optimizer.zero_grad()
    loss_history.append(loss)

Hi, it seems that the checkpoint files in huggingface are all used for class Model1d, I wonder if there exist any checkpoints available for the DiffusionAutoencoder1d class to perform the latent encoding?

Hello,

I upgraded the trainer and audio diffusion to the latest releases. I am now getting this error when trying to run experiments:

[2022-10-19 09:47:41,538][main][INFO] - Instantiating model <main.module_base.Model>. Error executing job with overrides: ['exp=base_youtube_l_3.yaml'] Error locating target 'audio_diffusion_pytorch.VDistribution', see chained exception above. full_key: model.model.diffusion_sigma_distribution

Set the environment variable HYDRA_FULL_ERROR=1 for a complete stack trace.

I deleted the conda environment and reinstalled all of the requirements from scratch, and I am still getting the above. Any help would be appreciated.

Thanks, MP

Two very simple fixes:

1. Fixes ValueError: num_channels must be divisible by num_groups when constructing stereo diffusion autoencoders.
2. Also fixes non-descript Torch Dynamo failure due to einops-exts (I have an issue open here)

I am trying to use the package to work with spectrograms, but I have encountered a problem. Some of the operations in the package are only designed to work with 3-d tensors, which limits their usability.

Request

I would like to request a change to make these operations more generic, so that they can be used with spectrograms (or any other data that may not necessarily be 3-d tensors). This would enable more users to use the package for a wider range of applications, and improve the overall usability of the package.

Examples

To illustrate the issue and the desired change, I have provided some examples below.

Sequential mask generation

The sequential_mask operation generates a boolean mask for a tensor. The original version of the operation is shown below:

def sequential_mask(like: Tensor, start: int) -> Tensor:
    length, device = like.shape[2], like.device
    mask = torch.ones_like(like, dtype=torch.bool)
    mask[:, :, start:] = torch.zeros((length - start,), device=device)
    return mask

To make this operation more generic, we could change the third dimension (dim=2) to the last dimension (dim=-1). This would allow the operation to work with any tensor, regardless of its shape. The revised version of the operation would look like this:

def sequential_mask(like: Tensor, start: int) -> Tensor:
    length, device = like.shape[-1], like.device
    mask = torch.ones_like(like, dtype=torch.bool)
    mask[..., start:] = torch.zeros((length - start,), device=device)
    return mask

I am happy to contribute, to address these issues.

TransformerBlock now uses Flash Attention when use_rel_pos isn't needed.

Note: I only tested this briefly with an untrained model for obvious crashes, and will continue with actual testing once I finish building a new training loop for my models. It works, but the results might not be correct.