Of course, you can use audioFLux to extract features and then build and train models for the sound effects audio that needs to be detected.

Then, real-time audio features are extracted from the audio stream obtained by the microphone, and a trained model is used for prediction.

It's awesome. Thank you for your work.

I'd like to know why you didn't take the LoRA approach to finetuning LLaMA? Is a full finetuning better?

A dumb model gets 99% acc on 1% prevalence of disease, easily and correctly.

Be careful what you ask for, like acc not f1

Combining leakage, well known datasets, small test data sets, wrong metric ie accuracy on very unbalanced data, a and good data science choices otherwise , is not unexpected to see perfect accuracy. It certainly can be that accurate, but who cares given all these othher possible failings in. The analysis.

It's plausible. To see 100% accuracy esp on well studied datasets or small set of new data. On a 20 example test set in the wild i witnessed exactly this. 20: is super small. The rules forbid me from using more .

OpenSMILE is mainly used for emotion analysis and classification of audio, while audioFlux focuses on various feature extraction of audio , and is used to study various tasks in the audio field such as Classification, Separation, Music Information Retrieval(MIR) and ASR etc.

llama.cpp and alpaca.cpp (and also related projects like llama-rs) only use the CPU. So not only are you not getting the most out of your GPU, it's not getting used at all.

I have an old GPU with only 6GB so running larger models on GPU isn't practical for me. I haven't really looked at that aspect of it much. You could start here: https://rentry.org/llama-tard-v2

Keep in mind you will need to be pretty decent with technical stuff to be able to get it working based on those instructions even though they are detailed.

A gpt might be engineered to rea papers and report findings of common basic errors in analysis design like you found.

Probability calibration could be added later via telemetry revealing its level of accuracy of its own basic error classification s.

There are quantized versions at 8bit and 4bit. The 4bit quantized 30B version is 18GB so it will run on a machine with 32GB RAM.

The bigger the model, the more tolerant it seems to quantization so even 1bit quantized models are in the realm of possibility (would probably have to be something like a 120B+ model to really work).

[removed]

it's always important to approach any claim of 100% accuracy with a critical eye. Achieving 100% accuracy is nearly impossible in any practical dataset, and it is usually an indication of overfitting or other statistical biases in the model.

It is also essential to examine the data transformation and feature selection process used in the model as these can have a significant impact on model performance and biases. It's important to ensure that these processes are transparent, unbiased, and validated using appropriate statistical methods.

Thank you for your comment. I have not worked with ResNets before, and the paper I used as a basis erroneously stated that they chose this architecture because of vanishing gradients. Wikipedia has the same error it seems.

Indeed, I am working with WGAN-GP. Unfortunately, implementing layer norm, while enabling me to scale the depth, completely changes the training dynamics. Training both G and C with the same learning rate and the same schedule (1:1), the critic seems to win, a behaviour I have never seen before in GANs. I suppose I will have to retune learning rates.

:facepalm:

Yeah, that is exactly the level of mistakes I have to deal with.

Another classic that I see repeated over and over again is wildly unbalanced datasets: Some diseases are very rare, so for every sample of the disease you are looking for, there are 10000 or more samples that are normal. And often, they just throw it into a classifier and hope for the best.

And then you can also easily get 99% accuracy, but the only thing the classifier has learned, is to say "normal tissue", regardless of the input.

So true - also I always think to the skin cancer detection model that turned out to predict anything with an arrow pointing to it to be cancer because all of the cancerous lesions in their training set had arrows. (Paper showing this ended up in JAMA)

I have the 7b 4bit alpaca.cpp running on my cpu (on virtualized Linux) and also this browser open with 12.3/16GB free. So realistically to use it without taking over your computer I guess 16GB of ram is needed. 8GB wouldn't cut it. I mean, it might fit in 8gb of system ram apparently, especially if it's running natively on Linux. But I haven't tried it. I tried to load the 13b and I couldn't.

Everyone is speaking about AI content, creative stories, texts.. but do companies or people in the real world actually use it for their products?

Claims of 100% accuracy always sets off alarm bells.

I do work in the medical field and the problem is that there are lots of physicians who want to make easy money: Start a startup, collect some data (which is easy for them), download some model they have read about but don't really understand and start training.

I work for a medical device manufacturer and sometimes have to evaluate startups. And the errors they make are sometimes so basic that it becomes clear that they don't have the first clue what they are doing.

One of those startups claimed 99% accuracy on ultrasound images. But upon closer inspection their product was worthless. Apparently they know that they needed to split their data into training/validation/test set.

So what did they do? They took the videos and randomly assigned frames to one of these sets. And since two consecutive frames are very similar to each other, of course you are going to get 99% accuracy. It just means absolutely nothing.

I don’t typically deal in breast cancer histopathology models but I do work with medical imaging full time as my day job - if I’m reading this correctly they use the Wisconsin Breast Cancer dataset (originally released in 1995!: https://archive.ics.uci.edu/ml/datasets/breast+cancer+wisconsin+(diagnostic))

First question - have breast cancer histopathology evaluation techniques changed since 1995? Checking out a quick lit review - yes: https://www.ncbi.nlm.nih.gov/pmc/articles/PMC8642363/#Sec2

So is this dataset likely to be useful today? Well… we don’t know the demographics of the population, we don’t know the split of severity of tumors in the population (this could be all easy cancers and not very generalizable/ useful to what someone sees on a day to day!), and the preprocessing required would need someone to take the digital image and extract all these features which honestly probably takes the same amount of time as the pathologist looking at the image and evaluating it. Also it sort of looks like they just used the features that came with the dataset…

They report the 100% accuracy on the training set and 99% on the testing set - great, theoretically any model can get to 100% accuracy on the training set so I almost always ignore this completely when papers do this unless there is a substantial drop off between training and testing or vice versa. But next question - are these results in line with similar published results on this particular dataset? Here’s an ARXIV paper from 2019 with similar results: https://arxiv.org/pdf/1902.03825.pdf

So nothing new here… it seems it’s possible and has been previously published to get 99% accuracy on this dataset…

Next question - is procedia a good journal? It publishes conference paper proceedings with an impact factor of 0.8 (kind of low). It’s unlikely this hit a rigorous peer review process, although I don’t like to throw our conference journals just because some of the big cool clinical trial results and huge breakthroughs are dumped in places like there. But in this case it seems like two researchers trying to get a paper out and not necessarily a ground breaking discovery (people have published on this dataset before and gotten 99% with random forest before!).

Final conclusion: meh.

could it be there is some tag within the image setted by doctors ?

Generally on medecine papers there is some sort of data leakage (like they do data augmentation before splitting to train, val and test)

It sounds too good to be true.

Ah I see, thank you for the clarification.

[deleted]

Looked fine at first glance, imo. I know nothing about medicine, but it’s nice to see that each of the models evaluated were upper 90% on accuracy.

I would want to see how the model performs on a much larger data set before trusting the validity.

The guy that wrote I have no mouth and I must scream has some amazing audio books, I would definitely love to hear him doing readings about some short stories